

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>Continuous Integration Architecture &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/css/custom.css" type="text/css" />

  
  
    <link rel="shortcut icon" href="../../_static/favicon.ico"/>
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
        <script src="../../_static/jquery.js"></script>
        <script src="../../_static/underscore.js"></script>
        <script src="../../_static/doctools.js"></script>
    
    <script type="text/javascript" src="../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../genindex/" />
    <link rel="search" title="Search" href="../../search/" />
    <link rel="next" title="资料库结构" href="../corpus/" />
    <link rel="prev" title="CephContext" href="../context/" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    

















<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="../../" class="icon icon-home"></a> &raquo;</li>
        
          <li><a href="../internals/">Ceph 内幕</a> &raquo;</li>
        
      <li>Continuous Integration Architecture</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
          
            <a href="../../_sources/dev/continuous-integration.rst.txt" rel="nofollow"> View page source</a>
          
        
      </li>
    
  </ul>

  
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../">
          

          
            
            <img src="../../_static/logo.png" class="logo" alt="Logo"/>
          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../start/intro/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../install/">安装 Ceph</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../cephadm/">Cephadm</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../rados/">Ceph 存储集群</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../cephfs/">Ceph 文件系统</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../radosgw/">Ceph 对象网关</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../architecture/">体系结构</a></li>
<li class="toctree-l1"><a class="reference internal" href="../developer_guide/">开发者指南</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../internals/">Ceph 内幕</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../blkin/">Tracing Ceph With LTTng</a></li>
<li class="toctree-l2"><a class="reference internal" href="../blkin/#tracing-ceph-with-blkin">Tracing Ceph With Blkin</a></li>
<li class="toctree-l2"><a class="reference internal" href="../bluestore/">BlueStore Internals</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cache-pool/">Cache pool</a></li>
<li class="toctree-l2"><a class="reference internal" href="../ceph_krb_auth/">如何配置好 Ceph Kerberos 认证的详细文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephfs-mirroring/">CephFS Mirroring</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephfs-reclaim/">CephFS Reclaim Interface</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephfs-snapshots/">CephFS 快照</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephx/">Cephx</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephx_protocol/">Cephx 认证协议详细阐述</a></li>
<li class="toctree-l2"><a class="reference internal" href="../config/">配置管理系统</a></li>
<li class="toctree-l2"><a class="reference internal" href="../config-key/">config-key layout</a></li>
<li class="toctree-l2"><a class="reference internal" href="../context/">CephContext</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="#">Continuous Integration Architecture</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#ceph-pull-requests">ceph-pull-requests</a></li>
<li class="toctree-l3"><a class="reference internal" href="#build-dependencies">build dependencies</a></li>
<li class="toctree-l3"><a class="reference internal" href="#uploading-dependencies">Uploading Dependencies</a></li>
<li class="toctree-l3"><a class="reference internal" href="#update-install-deps-sh">Update <code class="docutils literal notranslate"><span class="pre">install-deps.sh</span></code></a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../corpus/">资料库结构</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cpu-profiler/">Oprofile 的安装</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cxx/">C++17 and libstdc++ ABI</a></li>
<li class="toctree-l2"><a class="reference internal" href="../deduplication/">去重</a></li>
<li class="toctree-l2"><a class="reference internal" href="../delayed-delete/">CephFS delayed deletion</a></li>
<li class="toctree-l2"><a class="reference internal" href="../dev_cluster_deployement/">开发集群的部署</a></li>
<li class="toctree-l2"><a class="reference internal" href="../dev_cluster_deployement/#id5">在同一机器上部署多套开发集群</a></li>
<li class="toctree-l2"><a class="reference internal" href="../development-workflow/">开发流程</a></li>
<li class="toctree-l2"><a class="reference internal" href="../documenting/">为 Ceph 写作文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../encoding/">序列化（编码、解码）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../erasure-coded-pool/">纠删码存储池</a></li>
<li class="toctree-l2"><a class="reference internal" href="../file-striping/">File striping</a></li>
<li class="toctree-l2"><a class="reference internal" href="../freebsd/">FreeBSD Implementation details</a></li>
<li class="toctree-l2"><a class="reference internal" href="../generatedocs/">Ceph 文档的构建</a></li>
<li class="toctree-l2"><a class="reference internal" href="../health-reports/">Health Reports</a></li>
<li class="toctree-l2"><a class="reference internal" href="../iana/">IANA 号</a></li>
<li class="toctree-l2"><a class="reference internal" href="../kubernetes/">Hacking on Ceph in Kubernetes with Rook</a></li>
<li class="toctree-l2"><a class="reference internal" href="../libs/">库体系结构</a></li>
<li class="toctree-l2"><a class="reference internal" href="../logging/">集群日志的用法</a></li>
<li class="toctree-l2"><a class="reference internal" href="../logs/">调试日志</a></li>
<li class="toctree-l2"><a class="reference internal" href="../macos/">在 MacOS 上构建</a></li>
<li class="toctree-l2"><a class="reference internal" href="../messenger/">Messenger notes</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mon-bootstrap/">Monitor bootstrap</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mon-elections/">Monitor Elections</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mon-on-disk-formats/">ON-DISK FORMAT</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mon-osdmap-prune/">FULL OSDMAP VERSION PRUNING</a></li>
<li class="toctree-l2"><a class="reference internal" href="../msgr2/">msgr2 协议（ msgr2.0 和 msgr2.1 ）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../network-encoding/">Network Encoding</a></li>
<li class="toctree-l2"><a class="reference internal" href="../network-protocol/">网络协议</a></li>
<li class="toctree-l2"><a class="reference internal" href="../object-store/">对象存储架构概述</a></li>
<li class="toctree-l2"><a class="reference internal" href="../osd-class-path/">OSD class path issues</a></li>
<li class="toctree-l2"><a class="reference internal" href="../peering/">互联</a></li>
<li class="toctree-l2"><a class="reference internal" href="../perf/">Using perf</a></li>
<li class="toctree-l2"><a class="reference internal" href="../perf_counters/">性能计数器</a></li>
<li class="toctree-l2"><a class="reference internal" href="../perf_histograms/">Perf histograms</a></li>
<li class="toctree-l2"><a class="reference internal" href="../placement-group/">PG （归置组）说明</a></li>
<li class="toctree-l2"><a class="reference internal" href="../quick_guide/">开发者指南（快速）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../rados-client-protocol/">RADOS 客户端协议</a></li>
<li class="toctree-l2"><a class="reference internal" href="../rbd-diff/">RBD 增量备份</a></li>
<li class="toctree-l2"><a class="reference internal" href="../rbd-export/">RBD Export &amp; Import</a></li>
<li class="toctree-l2"><a class="reference internal" href="../rbd-layering/">RBD Layering</a></li>
<li class="toctree-l2"><a class="reference internal" href="../release-checklists/">Release checklists</a></li>
<li class="toctree-l2"><a class="reference internal" href="../release-process/">Ceph Release Process</a></li>
<li class="toctree-l2"><a class="reference internal" href="../seastore/">SeaStore</a></li>
<li class="toctree-l2"><a class="reference internal" href="../sepia/">Sepia 社区测试实验室</a></li>
<li class="toctree-l2"><a class="reference internal" href="../session_authentication/">Session Authentication for the Cephx Protocol</a></li>
<li class="toctree-l2"><a class="reference internal" href="../testing/">测试笔记</a></li>
<li class="toctree-l2"><a class="reference internal" href="../versions/">Public OSD Version</a></li>
<li class="toctree-l2"><a class="reference internal" href="../vstart-ganesha/">NFS CephFS-RGW Developer Guide</a></li>
<li class="toctree-l2"><a class="reference internal" href="../wireshark/">Wireshark Dissector</a></li>
<li class="toctree-l2"><a class="reference internal" href="../zoned-storage/">Zoned Storage Support</a></li>
<li class="toctree-l2"><a class="reference internal" href="../osd_internals/">OSD 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mds_internals/">MDS 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../radosgw/">RADOS 网关开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../ceph-volume/">ceph-volume 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../crimson/">Crimson developer documentation</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <div class="section" id="continuous-integration-architecture">
<h1>Continuous Integration Architecture<a class="headerlink" href="#continuous-integration-architecture" title="Permalink to this headline">¶</a></h1>
<p>In Ceph, we rely on multiple CI pipelines in our development. Most of these pipelines
are centered around Jenkins. And their configurations are generated using <a class="reference external" href="https://docs.openstack.org/infra/jenkins-job-builder/">Jenkins Job Builder</a>.</p>
<p>Let’s take the <code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">check</span></code> performed by Jenkins as an example.</p>
<div class="section" id="ceph-pull-requests">
<h2>ceph-pull-requests<a class="headerlink" href="#ceph-pull-requests" title="Permalink to this headline">¶</a></h2>
<p><code class="docutils literal notranslate"><span class="pre">ceph-pull-requests</span></code> is a jenkins job which gets triggered by a GitHub pull
request or a trigger phrase like:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">jenkins</span> <span class="n">test</span> <span class="n">make</span> <span class="n">check</span>
</pre></div>
</div>
<p>There are multiple parties involved in this jenkins job:</p>
<div class="graphviz"><object data="../../_images/graphviz-74adfafc74ca7f02ca74f00a20f820133bb21890.svg" type="image/svg+xml" class="graphviz">
<p class="warning">digraph  {
  rankdir=&quot;LR&quot;;
  github [
   label=&quot;&lt;git&gt; git_repo | &lt;webhooks&gt; webhooks | &lt;api&gt; api&quot;;
   shape=record;
   href=&quot;https://github.com/ceph/ceph&quot;;
  ];
  subgraph cluster_lab {
    label=&quot;Sepia Lab&quot;;
    href=&quot;https://wiki.sepia.ceph.com/doku.php&quot;;
    shape=circle;
    apt_mirror [
      href=&quot;http://apt-mirror.front.sepia.ceph.com&quot;;
    ];
    shaman [
      href=&quot;https://shaman.ceph.com&quot;;
    ];
    chacra [
      peripheries=3;
      href=&quot;https://chacra.ceph.com&quot;;
    ];
    subgraph cluster_jenkins {
      label=&quot;jenkins&quot;;
      href=&quot;https://jenkins.ceph.com&quot;;
      jenkins_controller [ label = &quot;controller&quot; ];
      jenkins_agents [ label = &quot;agents&quot;, peripheries=3 ];
    };
  };
  {
    rank=same;
    package_repos [ peripheries=3 ];
    pypi;
    npm;
  }
  github:webhooks -&gt; jenkins_controller [ label = &quot;notify&quot;, color = &quot;crimson&quot; ];
  jenkins_controller -&gt; jenkins_agents [ label = &quot;schedule jobs&quot; ];
  jenkins_agents -&gt; github:git [ label = &quot;git pull&quot; ];
  jenkins_agents -&gt; shaman [ label = &quot;query for chacra repo URL&quot; ];
  jenkins_agents -&gt; chacra [ label = &quot;pull build dependencies&quot; ];
  jenkins_agents -&gt; package_repos [ label = &quot;pull build dependencies&quot; ];
  jenkins_agents -&gt; pypi [ label = &quot;pull Python packages&quot; ];
  jenkins_agents -&gt; npm [ label = &quot;pull JavaScript packages&quot; ];
  jenkins_agents -&gt; apt_mirror [ label = &quot;pull build dependencies&quot; ];
  jenkins_agents -&gt; github:api [ label = &quot;update&quot;, color = &quot;crimson&quot; ];
}</p></object></div>
<p>Where</p>
<dl class="simple">
<dt>Sepia Lab</dt><dd><p><a class="reference external" href="https://wiki.sepia.ceph.com/doku.php">Sepia Lab</a> is a test lab used by the Ceph project. This lab offers
the storage and computing resources required by our CI infra.</p>
</dd>
<dt>Jenkins agents</dt><dd><p>are a set of machines which perform the CI jobs. In this case, they</p>
<ol class="arabic simple">
<li><p>pull the git repo from GitHub and</p></li>
<li><p>rebase the pull request against the latest master</p></li>
<li><p>set necessary environment variables</p></li>
<li><p>run <code class="docutils literal notranslate"><span class="pre">run-make-check.sh</span></code></p></li>
</ol>
</dd>
<dt>Chacra</dt><dd><p>is a server offering RESTful API allowing the clients to store and
retrieve binary packages. It also creates the repo for uploaded
packages automatically. Once a certain repo is created on chacra, the
configured shaman server is updated as well, then we can query shaman
for the corresponding repo address. Chacra not only hosts Ceph packages,
it also hosts quite a few other packages like various build dependencies.</p>
</dd>
<dt>Shaman</dt><dd><p>is a server offering RESTful API allowing the clients to query the
information of repos hosted by chacra nodes. Shaman is also known
for its <a class="reference external" href="https://shaman.ceph.com">Web UI</a>. But please note, shaman does not build the
packages, it justs offers information of the builds.</p>
</dd>
</dl>
<p>As the following shows, <a class="reference external" href="https://github.com/ceph/chacra">chacra</a> manages multiple projects whose metadata
are stored in a database. These metadata are exposed via Shaman as a web
service. <a class="reference external" href="https://github.com/ceph/chacractl">chacractl</a> is a utility to interact with the <a class="reference external" href="https://github.com/ceph/chacra">chacra</a> service.</p>
<div class="graphviz"><object data="../../_images/graphviz-b380fdf91c4b577ed878694dc8c0c0f1d455fd56.svg" type="image/svg+xml" class="graphviz">
<p class="warning">digraph  {
  libboost [
    shape=cylinder;
  ];
  libzbd [
    shape=cylinder;
  ];
  other_repos [
    label=&quot;...&quot;;
    shape=cylinder;
  ];
  postgresql [
    shape=cylinder;
    style=filled;
  ]
  shaman -&gt; postgresql;
  chacra -&gt; postgresql;
  chacractl -&gt; chacra;
  chacra -&gt; libboost;
  chacra -&gt; libzbd;
  chacra -&gt; other_repos;
}</p></object></div>
</div>
<div class="section" id="build-dependencies">
<h2>build dependencies<a class="headerlink" href="#build-dependencies" title="Permalink to this headline">¶</a></h2>
<p>Just like lots of other software projects, Ceph has both build-time and
run-time dependencies. Most of time, we are inclined to use the packages
prebuilt by the distro. But there are cases where</p>
<ul class="simple">
<li><p>the necessary dependencies are either missing in the distro, or</p></li>
<li><p>their versions are too old, or</p></li>
<li><p>they are packaged without some important feature enabled.</p></li>
<li><p>we want to ensure that the version of a certain runtime dependency is
identical to the one we tested in our lab.</p></li>
</ul>
<p>No matter what the reason is, we either need to build them from source, or
to package them as binary packages instead of using the ones shipped by the
distro. Quite a few build-time dependencies are included as git submodules,
but in order to avoid rebuilding these dependencies repeatedly, we pre-built
some of them and uploaded them to our own repos. So, when performing
<code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">check</span></code>, the building hosts in our CI just pull them from our internal
repos hosting these packages instead of building them.</p>
<p>So far, following packages are prebuilt for ubuntu focal, and then uploaded to
<a class="reference external" href="https://github.com/ceph/chacra">chacra</a>:</p>
<dl>
<dt>libboost</dt><dd><p>packages <a class="reference external" href="https://www.boost.org">boost</a>. The packages’ names are changed from <code class="docutils literal notranslate"><span class="pre">libboost-*</span></code> to
<code class="docutils literal notranslate"><span class="pre">ceph-libboost-*</span></code>, and they are instead installed into <code class="docutils literal notranslate"><span class="pre">/opt/ceph</span></code>, so
they don’t interfere with the official <code class="docutils literal notranslate"><span class="pre">libboost</span></code> packages shipped by
distro. Its build scripts are hosted at <a class="reference external" href="https://github.com/tchaikov/ceph-boost">https://github.com/tchaikov/ceph-boost</a>.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><style type="text/css">
span.prompt1:before {
  content: "$ ";
}
</style><span class="prompt1">tar xjf boost_1_76_0.tar.bz2</span>
<span class="prompt1">git clone https://github.com/ceph/ceph-boost</span>
<span class="prompt1">cp -ra ceph-boost/debian boost_1_76_0/</span>
<span class="prompt1"><span class="nb">export</span> <span class="nv">DEB_BUILD_OPTIONS</span><span class="o">=</span><span class="s1">&#39;parallel=6 nodoc&#39;</span></span>
<span class="prompt1">dpkg-buildpackage -us -uc -b</span>
</pre></div></div></dd>
<dt>libzbd</dt><dd><p>packages <a class="reference external" href="https://github.com/westerndigitalcorporation/libzbd">libzbd</a> . The upstream libzbd includes debian packaging already.</p>
</dd>
<dt>libpmem</dt><dd><p>packages <a class="reference external" href="https://github.com/pmem/pmdk">pmdk</a> . Please note, <code class="docutils literal notranslate"><span class="pre">ndctl</span></code> is one of the build dependencies of
pmdk, for an updated debian packaging, please see
<a class="reference external" href="https://github.com/ceph/ceph-ndctl">https://github.com/ceph/ceph-ndctl</a> .</p>
</dd>
</dl>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>please ensure that the package version and the release number of the
packaging are properly updated when updating/upgrading the packaging,
otherwise it would be difficult to tell which version of the package
is installed. We check the package version before trying to upgrade
it in <code class="docutils literal notranslate"><span class="pre">install-deps.sh</span></code>.</p>
</div>
<p>But in addition to these libraries, <code class="docutils literal notranslate"><span class="pre">ceph-mgr-dashboard</span></code>’s frontend uses lots of
JavaScript packages. Quite a few of them are not packaged by distros. Not to
mention the trouble of testing different combination of versions of these
packages. So we decided to include these JavaScript packages in our dist tarball
using <code class="docutils literal notranslate"><span class="pre">make-dist</span></code>.</p>
<p>Also, because our downstream might not want to use the prepackaged binaries when
redistributing the precompiled Ceph packages, we also need to include these
libraries in our dist tarball. They are</p>
<ul class="simple">
<li><p>boost</p></li>
<li><p>liburing</p></li>
<li><p>pmdk</p></li>
</ul>
<p><code class="docutils literal notranslate"><span class="pre">make-dist</span></code> is a script used by our CI pipeline to create dist tarball so the
tarball can be used to build the Ceph packages in a clean room environmet. When
we need to upgrade these third party libraries, we should</p>
<ul class="simple">
<li><p>update the CMake script</p></li>
<li><p>rebuild the prebuilt packages and</p></li>
<li><p>update this script to reflect the change.</p></li>
</ul>
</div>
<div class="section" id="uploading-dependencies">
<h2>Uploading Dependencies<a class="headerlink" href="#uploading-dependencies" title="Permalink to this headline">¶</a></h2>
<p>To ensure that prebuilt packages are available by the jenkins agents, we need to
upload them to either <code class="docutils literal notranslate"><span class="pre">apt-mirror.front.sepia.ceph.com</span></code> or <a class="reference external" href="https://github.com/ceph/chacra">chacra</a>. To upload
packages to the former would require the help our our lab administrator, so if we
want to maintain the package repositories on regular basis, a better choice would be
to manage them using <a class="reference external" href="https://github.com/ceph/chacractl">chacractl</a>. <a class="reference external" href="https://github.com/ceph/chacra">chacra</a> represents packages repositories using
a resource hierarchy, like:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">&lt;</span><span class="n">project</span><span class="o">&gt;/&lt;</span><span class="n">branch</span><span class="o">&gt;/&lt;</span><span class="n">ref</span><span class="o">&gt;/&lt;</span><span class="n">distro</span><span class="o">&gt;/&lt;</span><span class="n">distro</span><span class="o">-</span><span class="n">version</span><span class="o">&gt;/&lt;</span><span class="n">arch</span><span class="o">&gt;</span>
</pre></div>
</div>
<p>In which:</p>
<dl class="simple">
<dt>project</dt><dd><p>in general, it is used for denoting a set of related packages. For instance,
<code class="docutils literal notranslate"><span class="pre">libboost</span></code>.</p>
</dd>
<dt>branch</dt><dd><p>branch of project. This mirrors the concept of a Git repo.</p>
</dd>
<dt>ref</dt><dd><p>a unique id of a given version of a set packages. This id is used to reference
the set packages under the <code class="docutils literal notranslate"><span class="pre">&lt;project&gt;/&lt;branch&gt;</span></code>. It is a good practice to
version the packaging recipes, like the <code class="docutils literal notranslate"><span class="pre">debian</span></code> directory for building deb
packages and the <code class="docutils literal notranslate"><span class="pre">spec</span></code> for building rpm packages, and use ths sha1 of the
packaging receipe for the <code class="docutils literal notranslate"><span class="pre">ref</span></code>. But you could also the a random string for
<code class="docutils literal notranslate"><span class="pre">ref</span></code>, like the tag name of the built source tree.</p>
</dd>
<dt>distro</dt><dd><p>the distro name for which the packages are built. Currently, following distros are
supported:</p>
<ul class="simple">
<li><p>centos</p></li>
<li><p>debian</p></li>
<li><p>fedora</p></li>
<li><p>rhel</p></li>
<li><p>ubuntu</p></li>
</ul>
</dd>
<dt>distro-version</dt><dd><p>the version of the distro. For instance, if a package is built on ubuntu focal,
the <code class="docutils literal notranslate"><span class="pre">distro-version</span></code> should be <code class="docutils literal notranslate"><span class="pre">20.04</span></code>.</p>
</dd>
<dt>arch</dt><dd><p>the architecture of the packages. It could be:</p>
<ul class="simple">
<li><p>arm64</p></li>
<li><p>amd64</p></li>
<li><p>noarch</p></li>
</ul>
</dd>
</dl>
<p>So, for example, we can upload the prebuilt boost packages to chacra like</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ls *.deb <span class="p">|</span> chacractl binary create <span class="se">\</span>
  libboost/master/099c0fd56b4a54457e288a2eff8fffdc0d416f7a/ubuntu/focal/amd64/flavors/default</span>
</pre></div></div></div>
<div class="section" id="update-install-deps-sh">
<h2>Update <code class="docutils literal notranslate"><span class="pre">install-deps.sh</span></code><a class="headerlink" href="#update-install-deps-sh" title="Permalink to this headline">¶</a></h2>
<p>We also need to update <code class="docutils literal notranslate"><span class="pre">install-deps.sh</span></code> to point the built script to the new
repo. Please refer to the <a class="reference external" href="https://github.com/ceph/ceph/blob/master/install-deps.sh">script</a>,
for more details.</p>
</div>
</div>



           </div>
           
          </div>
          <footer>
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
        <a href="../corpus/" class="btn btn-neutral float-right" title="资料库结构" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
        <a href="../context/" class="btn btn-neutral float-left" title="CephContext" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>
        &#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).

    </p>
  </div> 

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>